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ABSTRACT 

While carrying out formative assessment activities over social network services (SNS), it has 
been noted that personalized notifications have a high chance of "the important post getting 
lost" in the notification feed. In order to highlight this problem, this paper compares within a 
posttest only quasi-experiment, a total of 104 first year undergraduate students, all of which 
are prospective ICT teachers, in two groups. A formative assessment system in the ubiquitous 
learning context is delivered over an SNS in both groups. In the first group, the SNS has been 
used for the entire assessment task. In the second group, the questions have been delivered 
and responses were received over mobile phone "SMS" messages, while the SNS was used 
solely for providing feedback. The cases were compared in terms of voluntary participation 
rates and academic success. Both response rates and academic success have been significantly 
higher in the SMS group. When asked their reasons for not responding to questions; the SNS- 
only group frequently reported "not noticing the questions being sent". This may indicate a 
flaw in message design for using social networks as LMS's. Sensible use of push-messages is 
advised. 

Keywords: Ubiquitous learning, formative assessment, LMS design, social networking. 
INTRODUCTION 

The past two decades have introduced such rapid advancements in information technology 
that, many concepts in our daily lives have been redefined. The field of Educational Sciences is 
no exception and the recent advancements in IT have currently brought around an age of 
Learning Environments in Instructional Design (Warren, Lee, & Najmi, 2014). Numerous 
systems have been developed in the past decade to offer students appropriate and dynamic 
learning environments, including but not limited to, digital Learning Management Systems 
(LMS). 

The use of technology in delivering instructional material to the new generation, who are most 
often thought to have much different and open-minded perspectives towards the said 
technologies than their predecessors, is most often viewed as a highly plausible idea. This 
notion is backed by the the famous Digital Natives theory by Prensky (Prensky, 2001). 
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However, as far as education is concerned; the learners of today fall into the category of the 
Millenium Generation, who are considered to be digital natives only in their "zones of comfort" 
(Oh & Reeves, 2014) i.e. environments which are comprised of few highly popular platforms, 
including Social Networking Sites. This fact has led educational researchers to the question as 
to whether Social Networking Sites, where today's learners seem to use so frequently and 
efficiently, could be used as Learning Management Systems (Wang, Woo, Quek, Yang, & Liu, 
2012). As displayed by DiVall and Kirwin, who have compared statistics on the use of the 
popular social networking service named Facebook and a traditional LMS named Blackboard, 
it was found that students were more likely to generate and be exposed to educational content 
on the Facebook social network service than on the traditional LMS, Blackboard (DiVall & 
Kirwin, 2012). Thomsen and colleagues (2016) also reported that as far as higher education is 
concerned, students are more likely to prefer using their existing Social Networks as learning 
systems than they do platforms specifically tailored for the purpose, such as Moodle. 

Another rising trend in the past decade. Mobile learning has been defined by Crompton as 
"learning across multiple contexts, through social and content interactions, using personal 
electronic devices" (Crompton, 2013). Considering that most social network services are cross¬ 
platform applications that operate on the Web, it is a fact that they can much easily be 
accessed ubiquitously with all popular smart mobile devices of today. And indeed, given the 
technical capabilities and the current popularity of social networks, the implications for their 
use as learning management systems in not just conventional but ubiquitous learning 
scenarios can be noted. 

Typical usage purposes of Learning Management Systems have been reported to be Class 
Management (Registering, Enrolling, Displaying schedules, etc.). Communication (E-mail, 
Chat, Forums, etc.). Content Delivery (Learning Resources, Learning Object Repositories, 
Links) and last but not least; Assessment, both summative and formative (Coates, James, & 
Baldwin, 2005). Whereas, formative assesment has been described as assessment that is 
specifically intended to generate feedback on performance to improve and accelerate learning 
(Sadler, 1998). And as far as online formative assessment is concerned. Kibble has noted in his 
work that "voluntary participation in online formative quizzes is a sensitive predictor of 
student success" (Kibble, 2011). 

Altogether, these findings could be used to back the notion of implementing a formative 
assessment system in the ubiquitous context over a popular social network platform, which 
can be accessed via PCs and Smartphones alike, for delivering drill and practice questions and 
providing feedback to students. 

There have already been attempts to carry out formative assessment activities over social 
network services at an undergraduate level, but it turns out that although educational 
activities over the platform is welcomed by students, academics are "advised to rethink its 
deployment in ways that foster student engagement, interaction and collaboration in a more 
constructivist and effective learning environment" (Shraim, 2013) That is, ways to deploy 
formative assessment over a Social Networking Service (SNS) for positively affecting student 
performance need exploring. 

It has also been reported that various factors influence students' participation and interaction 
with social network services used as a learning management systems. Among concerning 
factors are, the lack of notifications associated with posts in public domains that fail to alert 
and catch the attention of respective users; which results in "a high chance of the important 
post getting lost somewhere in the news feed of the target audience" as displayed by research 
conducted on the popular Facebook social network service by Shridhar and colleagues 
(Shridhar, Gupta, & Shridhar, 2014) . The same study has also displayed that the students also 
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reported feelings of having their area of privacy invaded. These clearly indicate problems in 
message delivery and/or interaction design, implying that although social networks are 
popular among today's learners, notifications regarding course activities may get lost among 
notifications of a more personal nature and/or simply fail to go above the attention threshold 
of the recipient. 

Understanding Smartphone Notifications 

Notifications; which are visual, auditory or haptic alerts that seek to grab the attention of the 
user, play a vital part in users' interaction with content presented by their smartphones. As 
explained by Iqbal and Bailey (2010) a notification "represents the proactive delivery of 
information to a user and reduces the need to visually scan or repeatedly check an external 
information source". Within the research domain of Human Computer Interaction, there exists 
numerous studies pertaining to understanding the nature of how users interact with 
smartphone notifications. Most of the time, the goal of these research efforts is to get the 
message across to the user as effectively and with as little disruption/interruption as possible. 
Many factors contribute to achieving this goal, such as timeliness of notifications (Iqbal & 
Bailey, 2010), mode of sensory cue (Gallud & Tesoriero, 2015), perceived sense of urgency of 
the message (Gallud & Tesoriero, 2015), the nature of the relationship between the user and 
the sender of the message that triggered the notification (Mehrotra et al., 2016), the nature 
of the application that triggered the notification (Mehrotra et al., 2016). As such, it can be said 
that not all applications and messaging channels create the same effect of 
interruption/ignorance/annoyance in the user (Turner, Allen & Whitaker, 2015). 

Findings indicate that the greatest factor in a user's response to a notification is the "urgency" 
of the notification, and notifications create a greater sense of urgency in the following 
conditions: 

> When they are coming from messaging apps (as opposed to Social Media apps) 

> When they are sent by an actual person (as opposed to being generated by a 
computerized system) (Mehrotra et al., 2016). 

These show that relying on notifications generated by Social Media applications for notifying 
users of content may be troublesome. In addition to the social media notification feed being 
overly congested by notifications from non-educational activities; Pielot, Church and Oliveira 
(2014) have explained as a result of their study investigating mobile phone notifications, that 
"Increasing numbers of notifications, in particular from email and social networks, are 
correlated with negative emotions, such as stress and feeling overwhelmed". These, in turn, 
may cause students to ignore the delivered educational content, particularly content that 
requires user input. 

One probable solution that may address these problems could be to retrospectively re-design 
the message interface of the ubiquitous formative assessment system to operate over the 
relatively older and fundamental "SMS" (short message service) functionality found in all 
contemporary cellular phones of today, costly or not. The rationale is that, as an utterly basic 
personal messaging tool, SMS messages offer a complete one-to-one and private 
communication channel between the sender and recipient, enabling to directly address each 
member of the target audience and successfully alerting them (assuming that this channel of 
communication is not much preferred anymore after the introduction of more sophisticated 
mobile data networks). Moreover, the SMS was projected to be more successful in going above 
the recipient's attention threshold by creating a greater sense of urgency and importance 
compared to social network notifications; and that participation rates may increase with the 
inclusion of students who have mobile phones that are not in the "Smartphone" category (i.e. 
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phones that are incapable of conneting to mobile data networks). Although SMS costs money 
per message sent, charges per SMS have dropped greatly since the introduction of GSM mobile 
phones to the general public approximately two decades ago and most GSM carriers offer free 
bulk SMS packages as part of subscriber plans. It has also been shown by previous research 
that SMS is indeed a viable tool for delivering learning content to students in an ubiquitous 
learning context (Gasaymeh & Aldalalah, 2013). Moreover, using SMS in a learning scenario 
could mean including not only the students who can successfully connect to the Web with their 
Smartphones, but also students who may have a mobile phone but for one reason or the other 
may not have Web connectivity over such expensive devices. 

The successful delivery of mobile phone notifications seems to be a valid issue in Human 
Computer Interaction research, but a review of the literature reveals that this topic has been 
largely ignored in instructional design of ubiquitous learning activities. It might therefore be 
worth investigating whether preferring a notification scheme in educational applications to 
another might cause a different educational outcome. It is thought that the use of messages 
that directly address recipients from over a less congested channel -as opposed to social 
network notification systems and in the form of SMS messages within the context of this 
research paper- for a formative assessment application has a potential to attract more 
students; which could then translate into higher academic success. An experimental setup has 
been considered for answering this question and regardless of what the outcome may be, it is 
thought to be important to raise the awareness of instructional design academic community 
towards the semi-technical aspect of user notifications that seems to have fallen under the 
attention of Human Computer Interaction practitioners, so far. 

Purpose 

The aim of the study is to determine whether directly addressing students with messages over 
a separate, less congested notifications channel to increase student engagement and academic 
success as opposed to relying on social network service notifications for notifying students of 
ubiquitous drill-and-practice based formative assessment activitiesin taking place in a social 
network. The rationale is that, although mobile applications of most social networks have 
push-notification capabilities for grabbing the attention of users, notifications concerning 
educational activities might get lost among the many other notifications of different contexts. 

In this context, the following research questions have been formulated considering two groups 
of of students; with one receiving and responding drill-and-practice questions over SMS and 
the other using social media (SNS) user groups for the drill-and-practice activity while relying 
on the built-in notification system: 

1. Which group will submit a greater number of responses to the drill and practice 
questions? 

2. Which group of students will show greater participation (measured by social 
network interaction units) to the "social phase" of the program, a phase included 
in both groups and involved peer discussion and instructor feedback over the social 
network? 

3. Which group of students will show greater academic success in terms of subjects 
covered by the drill and practice program? 

4. What will be the reasons for non-participation in the relevant drill and practice 
programs in each of the groups? 
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METHODOLOGY 


The study follows a posttest only quasi-experimental design with two groups. In order to 
provid additional control, the groups were matched and deemed non-inferior to one another 
with prior knowledge tests and surveys. The entire study group is comprised of 104 first year 
undergraduate students at Marmara University Ataturk Faculty of Education, Department of 
Computer Education and Instructional Technology; which are actually two classes of students 
(N1 = 49, N2 = 54) who are prospective information and communications technology (ICT) 
teachers. The two classes are actually daytime and evening education programs in the same 
department, and are taught the same content by the same instructor in the same subjects. For 
the purpose of this research, each of the classes are designated as separate groups with 
different interventions on each. 

The paper therefore seeks to compare two non-inferior groups in an intervention design where 
a formative assessment system on a popular Social Network Service is used for delivering drill- 
and-practice questions on a reference ICT subject of "SQL queries" to undergraduate students. 
In the first group's case, the entire system (comprised of the initial question delivery, response 
collection, peer discussion and instructor feedback) is handled over a popular social network 
platform, which notifies users of educational activities with its built-in notification system. 
Whereas in the second group, the question delivery and response collection phases were 
handled over SMS messages and the social network platform was used only for providing 
feedback to students regarding their activities over SMS. 

For the purpose of determining the type of intervention a particular group will receive, a 
purposive selection was made based on data from one Survey on Mobile Phone and SNS Usage, 
with the formula being the group that uses internet connection more often on their mobile 
phones to be designated as the "Social Network Only" group. The rationale has been to 
maximize the exposure of students to push-notifications, which can be received only from over 
mobile applications that operate on smart mobile devices with internet connectivity. 

Data Collection Instruments 

Data have been collected as part of the 1st year course of "Information Technologies in 
Education". The subject matter has been SQL queries. The reference social network service 
chosen for collection of data in both groups was Facebook, due to its popularity among 
students, whereas mobile phone SMS messages were chosen as a representation of messages 
directly addressing students via a non-congested channel. Furthermore, the following 
instruments have been used for collecting data that was compared between groups: 

Prior knowledge test on SQL queries 

The study began in the middle of the semester, while instruction was already taking place. This 
has proven to be a limitation, and to cover for reasons of group inequivalence that may arise 
due to different levels of understanding students in each group may have, a statistical method 
of comparison for non-inferiority of groups has been sought. A multiple choice test named 
"Prior knowledge test on SQL queries" has been used for this purpose. The test consisted of 
multiple-choice-only items measuring bulk knowledge pertaining to a limited subset of SQL 
commands. It was decided to exclude items requiring the use of a higher order cognitive skill, 
namely the formulation of SQL queries from scratch. The rationale has not been to assess 
student academic achievement (which is more related to query formulation), but only to see 
whether one group outperformed the other in terms of bulk knowledge. 

The test, which is comprised of 32 multiple choice questions, and scored over 100 points was 
used to determine equivalency, in the form of statistical non-inferiority, between groups in 
terms of academic pre-knowledge. The test was administered to 3rd year students at the same 
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undergraduate program in a different university and it was found to have an Internal 
consistency Cronbach's Alpha value of .917. 

Post-test on SQL queries 

This test has been used for comparing academic success between groups after the 
interventions. It measures command over SQL language for performing queries in a given 
database and is comprised of 15 Multiple Choice, as well as 10 open-ended query-formulation 
questions. Scored over 100 points, the test test was also administered to 3rd year students at 
the same undergraduate program in a different university and it was found to have an Internal 
consistency Cronbach's Alpha value of .835. 

Survey on mobile phone and SNS usage 

The survey has been used to find out about the mobile phone, mobile internet and social 
network usage habits of students. Comprised of items of ordinal measurement and open-ended 
questions. It has been used for determining the social network to be used within the study, as 
well as for determining which group will receive which intervention. It was been planned to 
designate the group that more frequently uses mobile internet connection as the "Social 
Network Only" experimental group, in order to maximize exposure to mobile push- 
notifications of mobile applications (as opposed to SMS notifications). 

Drill and practice questions 

16 drill and practice questions, delivered via Facebook group wall in Group 1 and via SMS 
messages to each student in Group 2, using a bulk text messaging software on a desktop PC 
connected over USB cable to a GSM mobile phone. The responses to these questions have been 
collected over the same Facebook wall as post comments in Group 1, and as SMS replies in 
Group 2. In each case, the students had a 3-hour time frame to respond the questions, at the 
end of which, correct answers were revealed to students as Facebook wall comments/posts in 
Group 1/Group 2, respectively. 

Several examples of the drill and practice questions on basic SQL queries, which were delivered 
to students from over their respective delivery channels, are as follows: 

Add an entry for the province of Yaiova, which lays in the region of Marmara and 
has a 850 km square land with 200000population, into the table Provinces (name, 
region, iandarea, population). 

Use the table Personnel (name, title, salary) to set the salaries of all employees 
with the Manager title as 4000 Turkish Liras. 

Use the table Grades (firstName, lastName, score, project) for rewarding a 10% 
increase in the score of all students who have contributed in a project (shown as 
"true"boolean property set in the project field). 

Use the Provinces (name, region, iandArea, population) table for listing provinces 
that have their names begin with letter A and that are located in the Mediterrean 
region. 

Facebook comments and likes 

Student interaction with the Facebook Wall of their relevant groups. The instructor has 
befriended each of the students in each group under a formal alias (not through a personal 
account). The data was interpreted and analyzed as the level of interaction over the social 
network in a given group, and used in answering research question 2. 
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Final survey 

Used for discovering the reasons behind student interaction with the practice program and the 
lack thereof. Comprised of open-ended questions, the content analysis of the survey has been 
carried out by one of the researchers. To provide for reliability of coding measures, the content 
has been coded by two researchers and the results have been compared to yield a good 
(Altman, 1990) Cohen's Kappa intercoder agreement rating of k = .762, p < .05. 

Instructor observation 

The instructor has operated the Facebook Groups through which the interventions have been 
carried out; sending questions, collecting responses and providing feedback. As such, she has 
befriended each of the students over the Facebook social network. The instructor's 
observations have thus been referred to when discussing the results for non-participation. 

The Intervention 

Once after the Pre-test on SQL queries and Survey on Mobile Phone and SNS Usage instruments 
have been administered, the groups' non-inferiority has been confirmed. This was followed by 
the choosing of the social network for use, which was "Facebook" as it was the most popularly 
used one among students. Depending on smartphone ownership data, it was decided which 
group would receive which intervention. 

The intervention thus began with 16 short answer practice questions on the subject of "SQL 
Queries" being delivered to both groups over the course of two weeks. It was made sure that 
at least one question was sent every day, and that both groups received their questions 
simultaneously. The time of sending questions varied randomly in the range of 1 PM to 4 PM 
for each day. This time range has been determined as the intersection of daily activity hours 
between groups, as one of the groups received their college classes after 5 PM as per after- 
hours education policy while the other group received classes normally during the daytime, 
starting at 9 AM. 

The intervention with Group 1 ("Facebook Only"), which received the questions on the wall of 
their relevant Facebook Group, has been explained in Figure la and detailed as follows: 

> Drill-and-practice questions are sent to the Facebook group as wall posts. 

> A time window (i.e. three hours) is given to the students for submitting their 
responses as comments to the wall post. 

> Instructor is notified of student activitiy through Facebook alerts and provides real¬ 
time monitoring and feedback for student responses. 

> At the end of the time window, the instructor shares the correct answer with the 
students as comment to the wall post, also providing feedback. 

Whereas, the intervention with Case 2 ("Facebook + SMS"), which received the questions on 
their mobile phones via SMS, has been explained in Figure lb and detailed as follows: 

> Drill-and-practice questions are sent to the students individually as SMS messages 
to their phones. 

> Students respond the questions directly by replying with SMS messages, within a 
given time window (i.e. three hours). 

> At the end of the time window, the instructor heads to Facebook wall and creates 
a wall post, revealing the question several hours ago, as well as the correct answer 
and the names of the respondents, in the chronological order of their relevant 
responses. 

> Feedback is provided by the instructor for any wrong answers as comments to the 
wall post. 
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Following the interventions, the academic success regarding the subject matter has been 
measured in each group with the Post-Test on SQL Queries. 


Group 1 - Facebook Only 


Group 2 - Facebook + SMS 
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Figure 1. The Groups and their relevant interventions explained 
(a) Facebook only group, (b) Facebook + SMS group. 


RESULTS 

Equivalence between Cases 

The equivalence between two cases in terms of subject matter (SQL Query) pre-knowledge 
and Facebook usage habits have been controlled for by administering and comparing the 
results of one SQL Query Skills pre-test and one question from the Survey on Mobile Phone 
and SNS usage, respectively. Independent sample comparisons have been carried out for 
statistically comparing the average scores from each set of data. The cases have been 
compared in terms of subject matter pre-knowledge by analysing their relevant SQL Query 
Skills pre-test scores. The independent samples non-parametric statistical comparison of non- 
normally distributed data obtained from each case have been shown in Table 1. 


Table 1. Mann-Whitney test results comparing 
the average scores in pre-test on SQL queries 


Group 

N 

Mean Rank 

Sum of Ranks U 

P 

Group 1 

49 

54,66 

2678,50 


Group 2 

55 

50,57 

2781,50 1241,50 

.46 


It was understood that there is no significant difference between groups in terms of SQL 
Queries pre-knowledge (Mann-Whitney U = 1241.50, p = .46). 

Choice of Social Network and Assignment of Groups to Interventions 

The cases have also been compared in terms of their social network usage habits by analysing 
the question "How often do you use the social network whose name has been given below" 
displayed in the Survey on Mobile Phone and SNS Usage. The response categories for the 
question have been [1] "Never heard of it", [2] "I only heard of it", [3] "I have an account that 
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I don't use", [4] "I rarely use it (one or two hours a week), [5] "I actively use it (Most days of 
the week, at least half an hour a day)", [6] "I use it very often (Every day, at least 3 hours a 
week)". The results have been detailed in Table 2. 


Table 2. Descriptive Statistics pertaining to the question "How often do you use the social 
network whose name has been given below" found in the in Survey on Mobile Phone and SNS 

Usage, scored over a scale of 1 to 6. 


Social 

Network 

Group 1 (N 

=49) 

Group 2(N 

=55) 

X 

S 

X 

S 

Facebook 

4.61 

0.99 

4.65 

0.93 

Twitter 

3.22 

1.10 

3.07 

1.41 

Google+ 

2.83 

1.00 

2.72 

0.78 

Friendfeed 

1.24 

0.43 

1.36 

0.55 

Reddit 

1.14 

0.35 

1.18 

0.54 

Delicious 

1.08 

0.27 

1.12 

0.38 

Tumblr 

1.32 

0.51 

1.51 

0.81 

Edmodo 

1.06 

0.24 

1.09 

0.29 


It was therefore understood that, the most frequently used social network for the students in 
both groups was Facebook, with average scores of 4.61 and 4.65 in Groups 1 and 2, 
respectively. It was therefore decided that Facebook would be used as the social network of 
choice for the study. 


In order to provide for additional control, the groups have been compared to determine 
whether there was a statistically significant difference between the distributions of their 
responses on how often they used the social network of choice for the research, Facebook. The 
independent samples non-parametric statistical comparison of non-normally distributed data 
obtained from two cases have been shown in Table 3. 

Table 3. Mann-Whitney test results comparing the average scores pertaining to Facebook 
obtained from the question "How often do you use the social network whose name has been 
given below" found in the in Survey on Mobile Phone and SNS Usage 


Group 

N 

X 

S 

Mean 

Rank 

Sum of 
Ranks 

U P 

Group 1 

49 

4.61 

0.99 

51,54 

2525,50 

1300,50 .74 

Group 2 

55 

4.65 

0.93 

53,35 

2934,50 



It was understood that the distribution of groups' answers to how often they used the 
Facebook social network did not differ from one another statistically (Mann-Whitney U = 
1300.50, p = .74). The groups have therefore been considered equivalent and comparable to 
one another in this context, assuming that initial habits for using Facebook would not alter the 
outcome of the research. 


The next step has been to determine which group would be treated with the Facebook Only 
intervention and which with the Facebook + SMS intervention. Examination of respones to the 
questions of "do you own a mobile phone" and "how often do you access the Internet using 
your mobile phone" in the Survey on Mobile Phone and SNS Usage have shown that 100% of 
students in both groups owned a cellular phone. As for frequency of Internet access from 
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mobile phones, students answered in a scale ranging from [1] Never, [2] "Sometimes", [3] 
"Often", [4] "Always". Distribution of scores in student responses has been given in Table 4. 


Table 4. Descriptive Statistics of Scores Showing Student Responses to the question 
"How often do you access the Internet using your mobile phone" 
found in the in Survey on Mobile Phone and SNS Usage 


Group 

N x 

S 

Group 1 

49 2,49 

1,05 

Group 2 

55 2,29 

0,91 


It was seen that Group 2 displayed a lower mean score for frequency in accessing the Internet 
from over the mobile phones as compared to Group 1. This has led to the decision of assigning 
Group 2 as the "Facebook + SMS" group and Group 1 as the "Facebook Only" group for 
convenience purposes. 

Number of Responses (RQ1) 

The bulk numbers of responses sent by students in each case have been compared for 
answering research question 1. Descriptive statistics on average number of submitted 
responses per student in groups, as well as the statistical comparison of the non-normally 
distributed data has been given in Table 5. 


Table 5. Mann-Whitney test results comparing 
the average number of responses in the Cases 




#of 

Responses 

X 



Mann-Whitney U 

Group 

N 

(responses 
per student) 

S 

Mean 

Rank 

Sum of ,, 

Ranks U p 

Facebook Only 

49 

13 

0,26 

1,19 

43,89 

2150,50 925,50 .00 

Facebook + SMS 

55 

92 

1,67 

2,79 

60,17 

3309,50 


It was seen that throughout the two week duration of the intervention, students in the 
Facebook + SMS group (N = 55, total # of responses = 92) have submitted a significantly 
larger number of responses over SMS, than the students in Facebook-only (N = 49, total # of 
responses = 13) group who were required to submit answers as comments to a Facebook wall 
post (Mann-Whitney U = 925.50, p = .00). 

In both groups, there were students that did not submit a single response throughout the 
intervention. Number of students that submitted at least one response has been found to be 
22 for the Facebook + SMS group (N = 55), whereas the same figure for the Facebook Only 
group was just 5. Figure 2 shows a stacked graph of number of responses, with each stack 
representing a single student's total number of responses. 
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Figure 2. The graph representation of number of responses 
per respondent and total number of respondents in each group. 

Visual examination of the graph in Figure 2 shows that the 92 responses are distributed more 
or less homogenously among the 22 respondents in the Facebook + SMS group, whereas 
majority of the few responses in the Facebook Only group has been submitted by a single 
student. 

Social Participation (RQ2) 

The amount of participation to the social phase of the program, which comprised the peer 
discussion and instructor component of the formative assessment program, has been 
measured by number of comments and likes to posts at each group's Facebook group wall. The 
results have been shown in Figure 3. 


# of Comments 


# of Likes 

0 2 4 6 8 10 12 14 16 

B Facebook Only H Facebook + SMS 

Figure 3. Number of Facebook Wall Comments 
(excluding comments that featured a response to the practice question) 
and Likes on Posts for each group during the drill and practice program. 

As it can be seen from the graph, both groups have shown relatively poor participation in the 
social phase during their two week practice runs. Still though, the activity in the Facebook + 
SMS group seems to be slightly higher compared to the Facebook Only group, due to the 
greater number of Likes. 

Academic Success (RQ3) 

At the end of their relevant two-week drill and practice programs, students at both case groups 
have taken a post-test that measures their academic success in the subject matter. The results 
of the independent samples t-test comparison of normally distributed data obtained from each 
case have been shown in Table 6. 
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Table 6. Student's T-test results comparing the average academic success 
in the Cases at the end of intervention 


Group 

N 


S 

df 

t 

P 

Facebook Only 

49 

45,18 

20,43 




Facebook + SMS 

55 

53,50 

17,75 

102 

-2,22 

.03 


It can be seen from the results that there is a significant difference between cases in terms of 
SQL Queries academic success following the drill-and-practice intervention in favor of the 
Facebook + SMS group, the members of which are more successful than those in the Facebook 
Only group (t(102) = -2.218, p = .03). 

Reasons for Non-Participation (RQ4) 

The final survey administered to each group at the end of the program was used for collecting 
student feedback. Among the questions of the survey, one open ended question urged the non¬ 
participating students to explain their reasons for doing so. The responses have been coded 
into categories and the results have been compared between groups. This has been shown in 
Figure 4. 


"I didn’t notice the questions being sent" 
"I don’t have regular access to the Internet 
"I already knew the subject matter well anyway" 
"I didn't want to befriend the instructor on Facebook" 
"I didn't receive the messages in my phone at all" 
"I wasn't around a PC when I received the messages" 
"I have no bulk SMS package" 
"I have no time for this" 
"I do not use Facebook much" 
"I didn't know the subject enough to send a response" 

"I wasn't interested" 
"I hesitated about sending in a wrong answer" 



0 2 4 6 8 10 12 14 


B Facebook + SMS H Facebook Only 


Figure 4. Coding categories of student responses in each group to the open-ended question 
"I did not participate in the practice program because..." 

It can be seen that, anxiety about answering wrong (Gl: 2 Codes, G2: 3 Codes), disinterest in 
the activity (Gl: 3 Codes, G2: 3 Codes) and lack of knowledge in the subject (Gl: 2 Codes, G2: 
2 Codes) have been reasons for non-response in both groups. Not being a frequent Facebook 
user has been another reason which was found in both groups, although this reason was much 
more prominent in Group 1 (13 codes vs. 4 in G2). However, when this last reported reason 
was compared with instructor observations of students' Facebook activities, it was inferred 
that student statements in this sense are not entirely true and possibly rather a simple excuse 
for evading the question.Other reasons found only within Group 1 were, knowing the subject 
matter too well (1 code), not wanting to befriend the instructor in Facebook (2 codes), not 
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having regular Internet access (2 codes) and last but not least, not noticing the questions 
being sent (11 codes). 

DISCUSSION 

This research has aimed to uncover whether messages directly addressing the learner from 
over a non-congested message channel could have a positive effect on student participation 
in formative assessment quizzes that are delivered over Social Networking Services in the 
ubiquitous-learning context, with the additional belief that results might provide insight in 
LMS design heuristics. A reference messaging system of "SMS" has been chosen to represent 
a non-congested direct-messaging medium due to convenience purposes and a quasi¬ 
experiment has been carried out with preservice ICT teachers as subjects. 

Although no initial significant differences were found between cases in terms of prior 
knowledge in the reference subject (SQL database queries) and Facebook usage frequencies; 
it has been shown that students who received the drill-and-practice questions over the 
Facebook wall have yielded significantly lower response rates compared to group of students 
who received questions in their mobile phones. This finding is highly consistent with results 
from a recent study by O'Bannon and her colleagues where a treatment group of students, who 
were asked to partake in educational activities over Facebook groups, have shown poor levels 
of participation and where no difference in terms of academic success between experimental 
and control groups have been observed (O'Bannon, Britt, & Beard, 2014). 

This situation could be explained with a simple allegory by likening the activity in the 
"Facebook Only" group to a teacher asking a question to the pupils to her students in a real- 
life classroom, where many factors -including but not limited to- shyness and anxiety could 
cause non-response. And just as clicker systems (Stowell, Oldham, & Bennett, 2010) that 
enable private communication between instructor and pupil are used in a real-world class, 
private messaging mediums, which is represented by SMS in the context of this study- could 
be used in online social environments to overcome peer pressure. 

At the end of the day, no matter how confident she may be, a student can still choose to be 
non-respondent if a question is not addressing her directly. And aside from dealing with peer- 
pressure, messages that directly address the learner may be helpful in this sense too, as 
addressed students may feel the immediate urge to respond, considering the instructor's full 
attention is now on her. 

Although the difference in the number of responses is almost tenfold between groups; the 
participation levels at the "social" phase of the formative assessment activity where students 
were expected to interact with each other or with the instructor's feedback through Facebook 
comments and likes, is not equally proportional. This could help reinforce the underlying fact 
within the interpretation of the previous question, i.e. students will not engage in an 
educational activity, especially in the social networking site, which they mostly use for 
leisurely activities, unless urged to do otherwise. However, it is important to note that the 
reason for low number of student comments in Group 2 could be due to the fact that, the use 
of Facebook wall in this group has been solely for the purpose of providing feedback to 
students at the end of 3-hour time windows throughout which they were expected to send 
responses over SMS. As such, most of the communication had taken place between student 
and instructor in private and when final feedback was provided over at the Facebook wall, 
there was not much else left to discuss. 
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As far as academic performance is concerned, it is also notable that the student group who has 
responded relatively well to the drill-and-practice questions over SMS has also shown 
significantly better academic performance, indicating that voluntary participation in formative 
assessment indeed is a predictor of success (Kibble, 2011) and that SMS is still a viable tool 
for use in formative assessments in the mobile-learning context (Cosgun & Ozdener, 2014). 

When asked their reasons for non-response, students have reported various reasons. Among 
these, the most striking contrasts were observed in terms of two reason categories. The first 
of these was was "I did not receive the questions being sent", which was highly pronounced 
in the Facebook Only group but not even once mentioned in the Facebook + SMS group. This 
finding was an actual manifestation of the claim that important social network service 
notifications may still go unnoticed in the recipient's news feed (Shridhar et al., 2014). The 
second important code of reasons for non-response has been the ambiguous statement of "I 
do not use Facebook much". Instructor observation of participating students' leisurely 
activities which comes into effect at this point however, shows through inference that this 
statement could be untrue and more or less an excuse to evade the question. At this point, it 
could be meaningful that none of the non-respondents at the Facebook + SMS group have 
thought to come up with the equivalent of such excuses. 

All and all, it can be inferred that students in this study have not shown much participation in 
the "social phases" of their practice activities that were carried out over the social networking 
service. 

CONCLUSION AND SUGGESTIONS 

It has been claimed in the literature that, as far as educational applications over social networking 
services in the ubiquitous learning context are concerned, there can be problems regarding the 
delivery of activity notifications to recipients. The significantly higher response rate and the 
consequential academic success achieved in this study by substituting public posts with direct 
messages that address learners personally over a non-congested channel could be considered to 
reinforce this claim. The SMS messaging system chosen in this study for convenience purposes, has 
been proven to be capable of serving as a personal messaging medium in the ubiquitous learning 
context in cases where the rate of smartphone ownership among students is not very high. As 
technology continues to become more and more accessible to the public, the SMS, which could 
already be perceived as an obsolete technology due to its usage costs and text limitations, could 
definitely be substituted with other forms of instant messaging. It should also be noted that a study 
by Amanullah and Ali reports that SMS, as opposed to contemporary messaging apps, has a higher 
rate of being congested by "spam" messages (Amanullah & Ali, 2014) and this can reduce user 
attention span. However, no matter what type of technology is used, the main idea is that leaving 
questions out in the open to be answered while using Social Networking Sites as Formative 
Assessment Tools in the ubiquitous learning context, may not be a best practice. 

It seems that the definitive results shown by this paper poses a dilemma: When planning to use a 
learning management system as a formative assessment tool in the ubiquitous context, one could 
always choose to use popular social networks for the job in order to harness their popularity but 
this means risking losing touch with the learners while notifications of leisurely posts get in the 
way. This risk was alleviated in the context of this paper by sending direct messages to learners 
through a communication channel that is considered non-congested. This approach also has 
implications in the way of LMS design in general, making sure once again that notifications 
regarding important activities that take place over the system should be successfully delivered to 
their recipients directly and through a non-congested channel. 

That being said, it should be noted that the "SMS" technology used in this study has the affordance 
of providing clear notifications to each cellular phone user, assuming that this relatively old 
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technology is almost unused for other purposes in the modern world. Considering the technical 
aspect of notification delivery in the ubiquitous context, notifications that are as powerful as SMS 
can be displayed only by native smartphone apps via their powerful feature of "push-notifications". 
Most Social Networking Services have highly popular native applications, but considering modern 
Learning Management Systems, one could not help but notice that most of these are applications 
can run only in Web browsers, which unfortunately render them unable to provide push 
notifications. There is always the option of using email-based push-notifications, but this could 
perhaps share the same fate with push notifications that are delivered by social network services: 
failing to go above the attention threshold of the recipient. This implies that future LMS designers 
might want to come up with native mobile application versions of their systems or at least 
complement their existing Web applications with SMS functionality, in order to provide for ubiquity 
in learning. 

Authors' Note: The research was funded and the equipment used in the study was made 
available by Marmara University Research Projects Board as part of research 
project number EGT-C-YLP-150513-0211. 
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